Coding schemes for virtual reality (VR) sequences

ABSTRACT

An improved method for coding video is provided that includes Virtual Reality (VR) sequences that enables more efficient encoding by organizing the VR sequence as a single 2D block structure. In the method, reference picture and subpicture lists are created and extended to account for coding of the VR sequence. To further improve coding efficiency, reference indexing can be provided for the temporal and spatial difference between a current VR picture block and the reference pictures and subpictures for the VR sequence. Further, because the reference subpictures for the VR sequence may not have the proper orientation once the VR sequence subpictures are organized into the VR sequence, reorientation of the reference subpictures is made so that the reference subpicture orientations match the current VR subpicture orientations.

CLAIM FOR PRIORITY

This Application is a continuation of U.S. patent application Ser. No.15/782,107 filed on Oct. 12, 2017, which claims priority under 35 U.S.C.§ 119(e) from earlier filed U.S. Provisional Patent Application No.62/407,108 filed on Oct. 12, 2016, all of which are incorporated hereinby reference in their entireties.

BACKGROUND Technical Field

The present invention relates to data structures used in coding VirtualReality (VR) streams using either Advanced Video Coding (AVC) or HighEfficiency Video Coding (HEVC or H-265). More particularly, the presentsystem relates to reference lists and indexing for reference picturesand subpictures used in coding VR pictures for AVC or HEVC.

Related Art

VR (Virtual Reality) is the term describing a three-dimensional,computer generated environment, which can be explored and interactedwith by a person. An example of use of VR is for 360 degree vision whichcould be achieved by special device with a Head Mounted Display (HMD) toenable a user to view all around. To cover the 360 degrees of vision inVR, a few projection formats have been proposed and used.

One VR format is cube projection which is illustrated using FIG. 1 . InFIG. 1 , a sphere is shown inside a cube to illustrate how the surfaceof the sphere can be projected out onto the surface of the cube. Cubeprojection maps can be used to project a map from a spherical globe outonto the surfaces of the cube. The map or other items on the sphere areprojected onto the six sides of the cube, each cube surface being a twodimensional square.

FIG. 2 illustrates the surfaces of the cube all provided onto a threedimensional surface. The surfaces of the cube in FIG. 2 are numbered toenable understanding conversion of the cube layout into the twodimensional layout of FIGS. 3 and 4 . In FIG. 3 , a 4×3 cube layout isshown, while in FIG. 4 a 3×2 layout is shown. The 4×3 and 3×2 cubelayouts of respective FIGS. 3 and 4 are basically the same, but withdifferent planar configuration for faces of the cube. In both FIGS. 3and 4 , the VR projection has 6 surfaces.

Two other VR formats other than the cube projection are described,although other formats might be used. One such VR format is the EqualRectangular Projection (ERP) which maps meridians of a map globe onto atwo dimensional surface with equally spaced vertical straight lines, andwith equally spaced horizontal straight lines. This enables longitudeand latitude lines on a globe to be equally spaced apart on the cube.Projection onto the surface of the cube still results in 6 surfaces thatcan be laid out as shown in FIGS. 3 and 4 .

Another VR format is the Equal Area Projection (EAP) which mapsmeridians of a map globe onto a two dimensional surface with equallyspaced vertical straight lines, and with circles of latitude mappeddirectly to horizontal lines even if they are not equally spaced. Again,projection onto the surface of the cube still results in 6 surfaces thatcan be laid out as shown in FIGS. 3 and 4 .

The existing video coding standards, such as Advanced Video Coding (AVC)or High Efficiency Video Coding (HEVC), may be used to code VRsequences. All those video coding standards are based upon a hybrid oftemporal and spatial coding. That is, the coding uses motion estimationand compensation (ME/MC) to remove the temporal redundancy betweenconsecutive pictures, and spatial prediction and spatial transform toremove the correlation among the pixels within a picture.

For ME/MC, the past-coded pictures are used as reference pictures forthe current and future pictures. A block in a current picture may find abest-matched (prediction) block in one or more reference pictures.Specifically, AVC and HEVC have two reference lists, which hold some ofthe past-coded pictures for future reference. A block in a currentpicture may find a prediction block in one of the pictures in each listof references.

It is desirable to provide improvements for coding when VR formats areused.

SUMMARY

Embodiments of the invention provide a method for coding video thatincludes VR sequences that enable more efficient encoding by organizingthe VR sequence as a single 2D block structure. Reference picture andsubpicture lists are created and extended to account for coding of theVR sequence. To further improve coding efficiency, reference indexingcan be provided for the temporal and spatial difference between acurrent VR picture block and the reference pictures and subpictures forthe VR sequence. Because the reference subpictures for the VR sequencemay not have the proper orientation once the VR sequence subpictures areorganized into the VR sequence, embodiments of the present inventionallow for reorientation of the reference subpictures so that thereference subpictures and VR subpictures are orientated the same.

For embodiments of the present invention, the VR sequence can be treatedas a regular 2D sequence. That is, each VR picture is treated as asingle 2D picture. In this case, all the existing video coding standardscan be applied to the single VR sequence directly. Since a VR picture ina cube of 4×3 or 3×2 includes six subpictures at each time instance, thesix subpictures. The six VR picture subpictures can be treated as sixtiles within a picture, similar to the concept defined in HEVC.

One embodiment of the present invention provides a method for coding ofvideo with VR pictures, with the coding including a reference list ofpast-coded pictures and subpictures. In the method, a current VR picturein the VR pictures of the video is defined to include six subpictures asrepresented by the cube of FIG. 3 . Next, at least one reference list isbuilt for the current VR picture, wherein the at least one referencelist holds a past-coded version of the VR picture as a reference pictureas well as the past-coded subpictures of the current VR picture asreference subpictures. Next, the reference list is divided into twoparts with the past-coded pictures provide in a first reference list.Past-coded subpictures are then provided in a second reference list.Next, motion vector prediction blocks are defined using the referencesubpictures from the first and second reference list for the current VRpicture. Finally, the motion vector prediction blocks are used in codingthat are also sent to the decoder.

Another embodiment of the present invention provides a method for codingof video with VR pictures that includes indexing of referencesubpictures relative to current subpictures to improve codingefficiency. In this embodiment also, a current VR picture in the VRpictures of the video is defined to include six subpictures. Next, areference picture and reference subpictures are defined for the currentVR picture. Then a reference list and index is built for the current VRpicture and subpictures relative to the reference picture andsubpictures. The indexing of subpictures is made according to temporaland spatial distances from a current block in one of the currentsubpictures to a reference block in the reference subpictures. Thereference list and index created is then used in coding of the video andsent to a decoder.

A further embodiment of the present invention provides a method forcoding of video with VR pictures that includes the ability to changesubpicture orientation to enable efficient encoding of the VR pictures.In this embodiment, like the embodiments above, a current VR picture inthe VR pictures of the video is defined to include six subpictures.Next, the subpictures for a reference picture for the current VR pictureis identified. Finally, the subpictures of the reference picture arerotated to match the orientation of the subpictures of the current VRpicture.

BRIEF DESCRIPTION OF THE DRAWINGS

Further details of the present invention are explained with the help ofthe attached drawings in which:

FIG. 1 illustrates how for a VR sequence a 360 degree spherical objectcan be mapped onto surfaces of a cube;

FIG. 2 illustrates the numbered surfaces of the cube that has beenmapped with a VR sequence from an internal spherical structure;

FIG. 3 shows organization of the cube surfaces of FIG. 2 into a 4×3 twodimensional structure for coding of the VR sequence;

FIG. 4 shows organization of the cube surfaces of FIG. 2 into a 3×2 twodimensional structure for coding of the VR sequence;

FIG. 5 provides a flowchart with steps according to embodiments of thepresent invention for coding video using reference picture andsubpicture lists to account for a VR sequence;

FIG. 6 provides a flowchart with steps indicating how referencesubpicture indexing is provided according to embodiments of the presentinvention;

FIG. 7 illustrates pictures used to create a reference list index with areference subpicture assigned a temporal index, i, and a spatial index,j;

FIG. 8 shows how the six subpictures in a reference picture are rotatedfor a current subpicture 2;

FIG. 9 shows how subpictures of a reference picture are rotated to havethe same orientation with the current picture ranging from subpicture 0through 5;

FIG. 10 provides a flowchart with steps showing how VR referencesubpicture orientation is changed so that the orientation matches thecurrent subpicture; and

FIG. 11 shows an encoder and decoder that can be configured to performencoding and decoding with VR pictures according to embodiments of thepresent invention.

DETAILED DESCRIPTION

A VR sequence in a video can be treated as a regular 2D sequence withsix subpictures for the embodiments of the invention described herein.That is, each VR picture is treated as a single 2D picture and codingstandards such as AVC and HEVC can be applied to the single VR sequencedirectly. The VR picture can be a 4×3 or 3×2 breakdown of a cube intosix subpictures at each time instance, as illustrated in FIGS. 3 and 4 .The six VR picture subpictures can be treated as six tiles within apicture, similar to the concept defined in HEVC.

To accomplish motion estimation and compensation (ME/MC) for embodimentsof the present invention, the concept of reference pictures lists,reference indexing and an orientation of references relative to acurrent picture can be provided for a VR sequence for embodiments of thepresent invention. A description of each of these concepts is providedto follow.

A. Reference Lists

The concept of reference pictures and lists can be extended for a VRsequence. Similar to AVC and HEVC, for a block in a current subpicturewithin a current picture, reference pictures can be provided andreference lists built to enable ME/MC. Reference pictures can be builtfrom the past-coded pictures of subpictures as well as the past-codedsubpictures of the current picture. A listing of these referencepictures can further be created.

The past-coded pictures can be included in at least one reference list,similar to AVC and HEVC. The past-coded subpictures for the currentpicture may be included in a second reference list.

Now for blocks, consider a current block in a current subpicture withina current picture. For the current block the reference prediction blockcan be found in one of the reference subpictures per reference list. Oneof reference subpictures in which the reference prediction block isfound can be in one of the past-coded pictures in a different picturetime instance than the current time instance forming the reference.

FIG. 5 provides a flowchart with steps according to embodiments of thepresent invention for coding video using reference picture andsubpicture lists to account for a VR sequence. In a first step 500, themethod defines a current VR picture provided in a video to have sixsubpictures. Next in a step 502, at least one reference list for thecurrent VR picture is built, wherein the at least one reference listholds a past-coded version of the VR picture as well as the past-codedsubpictures of the current VR picture. In step 504, the past-codedpictures are separated out into a first reference list. In step 506, thepast-coded subpictures are included in a second reference list. In step508, motion vector prediction blocks are defined using the referencesubpictures from the first and second reference list for the current VRpicture. Finally, in step 510 the motion vector prediction blocks areused in coding that is sent to the decoder.

B. Reference Indexing

Due to the fact that the closer the reference picture and subpicturesare to the current subpicture temporally and spatially, the higher thecorrelation between the reference picture and subpictures and thecurrent picture, the reference pictures and subpictures for embodimentsof the present invention may be indexed according to their temporal andspatial distance to the current subpicture.

Embodiments of the present invention provide for a default referencepicture/subpicture index order. In particular, for a current block in acurrent subpicture for a current picture, a reference picture andsubpictures in a reference picture list are indexed according to itstemporal and spatial distances to the current block in the currentsubpicture of the current picture. In other words, the closest referencepicture/subpicture to the current block in the current subpicture of thecurrent picture temporally and spatially is assigned the index of 0, thesecond closest reference picture/subpicture is assigned the index of 1,and so on.

FIG. 6 provides a flowchart with steps indicating how referencesubpicture indexing is provided according to embodiments of the presentinvention. In particular, the method illustrated by the flowchart ofFIG. 6 provides for coding a video containing virtual reality (VR)pictures that includes indexing of reference subpictures relative tocurrent subpictures. In a first step 600, a current VR picture in the VRpictures is defined to include six subpictures. Next in step 602, areference picture and reference subpictures for the current VR pictureis defined. In step 604 a reference list and index is built for thecurrent VR picture and subpictures relative to the reference picture andsubpictures. In step 606, indexing of the subpictures of the referencepicture to the subpictures in the current picture is provided accordingto temporal and spatial distances to a current block in a current one ofthe subpictures relative to a reference block in the referencesubpictures. Finally, in step 608, the reference list and index are usedin coding that is sent to the decoder.

In embodiment for providing a reference list index, a referencesubpicture is assigned a temporal index, i, and a spatial index, j, or acombination of temporal and spatial indexes, i+j. The temporal index, i,can be determined by the temporal distance between the reference pictureand the current picture, i.e., the closer, the smaller the index. Thespatial index, j, can be determined by the spatial distance between thereference subpicture in the reference picture and the current blockcollocated in the reference picture.

FIG. 7 illustrates pictures used to create a reference list index with areference subpicture assigned a temporal index, i, and a spatial index,j. In FIG. 7 , a current block 702 in gray color is shown in subpicture0 of a current picture 700. As seen, in the middle of FIG. 7 , theclosest subpicture to the collocated block 712 of the current block in areference picture 710 is subpicture 2. Hence, for the current block,subpicture 2 in any reference picture of any reference list will beassigned a spatial reference index of j=0. Subpicture 1 is the secondcloset subpicture, and so, it will be assigned the spatial referenceindex of j=1. For this example, for the current block in subpicture 0 ofthe current picture, the spatial reference indexes of j=0, 1, 2, 3, 4,and 5 will respectively be assigned to subpictures 2, 1, 4, 3, and 5 ofany reference picture of any reference list.

C. Subpicture Rotation

Not all the subpictures in a reference picture have the same orientationas the current subpicture of a current VR picture. To enable coding ofthe VR picture efficiently, the orientation of the six subpicturesmaking up the VR picture that is made up of arranged faces of a cubeshould be organized to have the same orientation irrespective ofarrangement of the cube faces. FIG. 8 shows how the six subpictures in areference picture are rotated for a current subpicture 2. A seen, inthis example, subpicture 1 needs to be rotated by 90 degreecounterclockwise, subpicture 4 to be rotated 90 degree counterclockwiseand subpicture 5 needs to be rotated by 180 degree clockwise. FIG. 9shows how subpictures of a reference picture are rotated to have thesame orientation with the current subpicture ranging from picture 0through 5.

Accordingly, embodiments of the present invention provide for thesubpictures of a reference picture to be rotated as shown in FIG. 9accordingly so that they can have the same orientation as the currentsubpicture, before any prediction is performed. FIG. 10 provides aflowchart with steps showing how VR reference subpicture orientation ischanged so that the orientation matches the current subpicture. In afirst step, 1000, a current VR picture in the VR pictures is defined toinclude six subpictures. Next in step 1002, subpictures for a referencepicture for the current VR picture are identified. In step 1004 acurrent subpicture of the current VR picture is identified. Finally, instep 1006 subpictures of the reference picture are oriented to match theorientation of the current subpicture of the current VR picture.

For better temporal and spatial prediction, the subpictures in areference picture are rotated and rearranged accordingly so that thespatial content transition from a subpicture to its neighbor subpictureswithin the reference picture can be continuous and smooth. It is notedthat in addition with rotation so that arrangement of subpictures of thecurrent and reference pictures are the same, the spatial referenceindex, j, may not be necessary as the reference picture of sixsubpictures can be treated as one single picture in the reference list.

FIG. 11 shows an encoder 102 and decoder 104 that can be configured toperform encoding and decoding with VR pictures according to embodimentsof the present invention. Motion estimation and motion compensation isperformed using information from embodiments of the present inventionwith encoder 102 and decoder 1104 using a process of determining amotion vector (MV) for a current unit of video. For example, the motionestimation process searches for a best match prediction for a currentunit block of video (e.g., a prediction block) over reference pictures.Motion compensation is then performed by subtracting a reference unitpointed to by the motion vector from the current unit of video.

To perform motion estimation and compensation, encoder 1102 and decoder1104 include motion estimation and compensation blocks 1104-1 and1104-2, respectively. For bi-directional prediction, the motionestimation and compensation blocks 1104-1 and 1104-2 can use a combinedbi-directional reference unit in the motion compensation process for thecurrent unit.

For the encoder 1102 and decoder 1104 of FIG. 11 , embodiments of thepresent invention contemplate that software to enable them to performfunctions described to follow for the present invention is provided in amemory. The encoder 1102 and decoder 1104 are further contemplated toinclude one or more processors that function in response to executablecode stored in the memory to cause the processor to perform thefunctions described.

Although the present invention has been described above withparticularity, this was merely to teach one of ordinary skill in the arthow to make and use the invention. Many additional modifications willfall within the scope of the invention as that scope is defined by thefollowing claims.

What is claimed:
 1. A method of coding a video containing virtualreality (VR) pictures that includes a reference list of past-codedpictures and subpictures, the method comprising: defining a current VRpicture in the VR pictures as a plurality of subpictures; building atleast one reference list for the current VR picture, wherein the atleast one reference list references at least one reference pictureselected from a temporally offset past-coded version of the current VRpicture as well as reference pictures selected from temporallyconcurrent past-coded subpictures of the current VR picture; includingthe temporally offset past-coded pictures in a first reference list;including the temporally concurrent past-coded subpictures in a secondreference list; defining motion vector prediction blocks using referencesubpictures from the first and second reference list for the current VRpicture; and using the motion vector prediction blocks in coding that issent to the decoder.
 2. The method of claim 1, further comprising:building a reference index for the current VR picture and subpicturesrelative to the reference picture and subpictures; indexing thesubpictures of the reference picture to the subpictures in the currentpicture according to temporal and spatial distances to a current blockin a current one of the subpictures to a reference block in thereference subpictures; and using the reference list and index in codingsaid video.
 3. The method of claim 2, wherein for the current block in acurrent picture, a reference subpicture is assigned a temporal index iand a spatial index j or a combination of temporal and spatial indexes,i+j, wherein the temporal index i can be determined by the temporaldistance between the reference picture and the current picture, andwherein the spatial index j can be determined by the spatial distancebetween the reference subpicture and the current subpicture block.
 4. Amethod of coding a video containing virtual reality (VR) pictures thatincludes indexing of reference subpictures relative to currentsubpictures, the method comprising: defining a current VR picture in theVR pictures as a plurality of subpictures; defining a reference picturetemporally offset from the current VR picture and reference subpicturestemporally concurrent with the current VR picture; building a referencelist and index for the current VR picture and subpictures relative tothe reference picture and subpictures; indexing subpictures of thereference picture to the subpictures in the current VR picture accordingto temporal and spatial distances to a current block in a current one ofthe subpictures; and using the reference list and index in coding saidvideo.
 5. The method of claim 4, wherein for the current block in acurrent picture, a reference subpicture is assigned a temporal index iand a spatial index j or a combination of temporal and spatial indexes,i+j, wherein the temporal index i can be determined by the temporaldistance between the reference picture and the current picture, andwherein the spatial index j can be determined by the spatial distancebetween the reference subpicture and the current subpicture block. 6.The method of claim 4, wherein a closest reference subpicture to thecurrent block in the current subpicture of the current picturetemporally and spatially is assigned the index of 0 in the referencepicture index, and the second closest reference subpicture is assignedthe index of 1 in the reference picture index.
 7. The method of claim 4,further comprising: identifying a current subpicture of the current VRpicture; and rotating the subpictures of the reference picture to matchthe orientation of the subpictures of the current VR picture.